Augmented Reality System, Method, and Apparatus for Displaying an Item Image in a Contextual Environment

ABSTRACT

Method, apparatus, and system for providing an item image to a client for display in a contextual environment are described. In some embodiments, the user may select an item for display in the contextual environment, and the user may position a camera coupled to a processing system to capture the contextual environment. A placeholder may be generated and associated with an item selected by a user. In an embodiment, the generated placeholder may be placed in a location within the contextual environment, and the user&#39;s processing system may send a visual data stream of the camera-captured environment to a server. In an embodiment, the user&#39;s processing device may receive a modified data stream including an image of the item, and the user&#39;s processing device may display the item image in the same location as the placeholder.

RELATED APPLICATIONS

This application is a continuation of and claims the benefit of priorityto U.S. patent application Ser. No. 14/868,105, filed Sep. 28, 2015,which claims priority under 35 U.S.C. § 120 to U.S. patent applicationSer. No. 12/644,957, filed on Dec. 22, 2009, the benefit of priority ofwhich is claimed hereby, and which is incorporated by reference hereinin its entirety.

BACKGROUND

Item listings on online shopping and auction websites (e.g., e-commercecompanies) may often provide images related to the item for sale tosupplement a text description of the item. The images may be in the formof a digital photograph, a drawing, or in some cases, a video clip orany combination of these. For some items, there may be multiple images(e.g., digital photos) providing different angles and views of the item.In some examples, the images may be in a contextual environment, such asa sofa in a living room. In other examples, the item image may merelydepict the item without a contextual environment.

BRIEF DESCRIPTION OF DRAWINGS

Embodiments of the present invention are illustrated by way of exampleand not limitation in the Figures of the accompanying drawings, in whichlike reference numbers indicate similar elements and in which:

FIG. 1 is a diagrammatic representation of a network within which anexample augmented reality system for providing modified visual data to aprocessing system may be implemented;

FIG. 2 is a block diagram of an example augmented reality module, inaccordance with one example embodiment;

FIG. 3 is a flow chart of a method for providing modified visual data,in accordance with an example embodiment;

FIG. 4 is an example of a placeholder for use with the augmented realitysystem, in accordance with an example embodiment;

FIGS. 5A-B are example user interfaces illustrating a placeholder in acamera-captured environment and the resulting augmented reality displayin a contextual environment, in accordance with an example embodiment;and

FIG. 6 is a diagrammatic representation of an example machine in theform of a computer system within which a set of instructions, forcausing the machine to perform any one or more of the methodologiesdiscussed herein, may be executed.

DETAILED DESCRIPTION

Systems and methods for displaying an item image on a screen as the itemwould appear positioned at a location in a camera-captured environmentare discussed. In the following description, for purposes ofexplanation, numerous specific details are set forth in order to providea thorough understanding of an embodiment of the present invention. Itwill be evident, however, to one skilled in the art that the presentinvention may be practiced without these specific details. The followingdetailed description includes reference to the accompanying drawings,which form a part of the detailed description. The drawings showillustrations in accordance with example embodiments. These exampleembodiments, which are also referred to herein as “examples,” aredescribed in enough detail to enable those skilled in the art topractice the present subject matter. The embodiments may be combined,other embodiments may be utilized, or structural, logical and electricalchanges may be made without departing from the scope of what is claimed.The following detailed description is, therefore, not to be taken in alimiting sense, and the scope is defined by the appending claims andtheir equivalents.

Augmented reality generally refers to creating a mixed virtual and realworld environment though computer-generated imagery. Elements of thereal world are mixed with virtual images in a real-time environment toprovide a user with information and a visual effect that may not beentirely available otherwise.

In an online publication system such as an online trading platform,where a seller may post items for sale, it may be desirable to view howa particular item would appear in a location specified by a user to aidthe user's purchase decision. For example, a user considering purchasinga sofa for a living room may need to determine where to position thesofa, and generally, how the sofa would appear in the living room. Theuser may wish to view the sofa as it would appear in the living roomwithout actually possessing the sofa. Furthermore, the user may preferto compare various sofas, view how the sofa would look from differentangles in the room, or even surround the sofa with other items to createa furniture ensemble without physically moving the actual furniture. Inan example embodiment, an augmented reality system may provide a userwith the ability to arrange and rearrange a virtual sofa (and additionalitems, if desired) in an environment (e.g., a living room) captured by acamera instead of requiring physical movement of an actual sofa.

In an example embodiment, the method allows a user to place a marker(e.g., a placeholder such as a piece of paper with contrasting markingsand unique objects) in a location where the actual item would bepositioned. The method records an image of the market positioned in thelocation using, e.g., a video camera, sends the captured image to theaugmented reality system, and receives an image of the actual itempositioned at the chosen location. In various embodiments, the marker(e.g., placeholder) can be a sheet of paper (8.5″×11″) with uniquemarkings and that a user can download, print and placed at the location.A server receives the video stream of the placeholder in the locationfrom the user's system, detects the placeholder, and sends a modifiedvideo stream with an item in place of the marker back to the user'ssystem for display.

In various example embodiments, an input including item selection isreceived from a client to generate a placeholder and information forgenerating the placeholder is provided to the input source (e.g., theclient) or a processing system with output capabilities. Video data of acamera-captured environment is received, in which the placeholder may belocated. The input is analyzed to detect the placeholder and an image isretrieved according to the analysis. The visual data is modified toinclude an image data associated with the image, which, in someembodiments, may be an overlay of the image, and the modified visualdata may be provided to the input source (e.g., a user's processingsystem) or another processing system. In some example embodiment, theretrieved image may be modified in accordance with the analysis to scaleand orient the item image overlay in accordance with the environment andplacement information determined from the placeholder information. Inone example embodiment, the placeholder may be a printed sheet of paperwith unique markings (e.g., bar code, contrastable image,two-dimensional data code) printable by the user from a home computersystem. In some example embodiments, the sheet of paper may also berecognizable by other processing systems. In some embodiments, a singleplaceholder may be generic to multiple items and used interchangeably torepresent more than one item. For example, when a user wishes to comparea first item with a second item in the same location, the placeholdermay interchangeably represent both the first and second item, withneeding to generate two placeholders. In some embodiments, a singleplaceholder can be used without associating the placeholder with anyparticular item. An example augmented reality system is discussed withreference to FIG. 1.

FIG. 1 is a diagrammatic representation of a network within which anexample augmented reality system 100 for providing modified visual datato a processing system 110 may be implemented. As shown, the augmentedreality system 100 may include a processing system 110 and a server 120.The server 120, in one example embodiment, may host an on-line tradingplatform. The processing system 110 may run an application 116, such asa web browser application and may have access to the server 120 via anetwork 130. The network 130 may be a public network (e.g., theInternet, a wireless network, etc.) or a private network (e.g., a localarea network (LAN), a wide area network (WAN), intranet, etc.).

In one embodiment, the network-based trading platform may provide one ormore marketplace applications, payment applications, and otherresources. The market place application may provide a number ofmarketplace functions and services to users that access the marketplace.The payment applications, likewise, may provide a number of paymentservices and functions to users. The network-based trading platform maydisplay various items listed on the trading platform. The embodimentsdiscussed in the specification are not limited to network-based tradingplatforms, however. In additional embodiments, other publishingplatforms, such as social networking websites, news aggregatingwebsites, web portals, network-based advertising platforms, or any othersystem that display items to users, may be used.

The processing system 110 may include a camera source 112 (e.g., a videocamera or digital still camera) for capturing a contextual environment(e.g., a living room) and a visual output 114 (e.g., an LCD display orthe like) to present the contextual environment and, in an exampleembodiment, to present an augmented environment. The server 120 includesan item image source 122 and an augmented reality application 124. Insome embodiments, the server 120 may be communicatively coupled with oneor more databases 140, which may store item images 142.

In an example embodiment, the processing system 110 may send a requestto the server 120 to modify a contextual environment captured by acamera source 112. In some embodiments, the request from the processingsystem 110 may include a user-selected item. In an embodiment, anaugmented reality application 124 at the server 120 may detect therequest, and provide placeholder generation information to theprocessing system 110 to generate a placeholder associated with theuser-selected item. Upon receiving the placeholder generationinformation, a user in control of the processing system 110 may producethe placeholder, e.g., by printing an image of the placeholder providedby the augmented reality application 124. In some example embodiments,the placeholder may be an ordinary letter-sized sheet of paper (e.g.,8.5″×11″) with unique markings, an example of which is illustrated inFIG. 4 and described below. The unique markings may be utilized by theaugmented reality application 124 to identify the placeholder in thecontextual environment and to determine a desired placement of theuser-selected item within the contextual environment captured by thecamera. In some embodiments, the placeholder may be generated using anauxiliary display device (e.g., mobile phone display) displaying theplaceholder with unique markings. In other embodiments, the user may usethe placeholder generation information to physically create theplaceholder by drawing and/or coloring the unique markings onto anobject, such as a piece of paper, or a three-dimensional object. In someembodiments, in addition to the user-selected item, the placeholdergeneration information and associated unique markings may be customizedto accommodate elements of the environment based on received visual datacollected by a camera. For example, the placeholder generationinformation may be produced by taking into considerations dimensions ofthe environment (e.g., room).

In another example embodiment, the augmented reality application 124 mayreceive and process visual data received from the processing system 110and the camera source 112 communicatively coupled to the processingsystem 110, and provides modified visual data back to the processingsystem 110. For example, the camera source 112 may collect visual dataof a contextual environment within which a placeholder is located. Insome embodiments, one of the applications 116 may receive visual data.The application 116 may provide the visual data to the augmented realityapplication 124 of the server 120 via network 130. The augmented realityapplication 124 may analyze the visual data to detect placeholderinformation associated with a placeholder located within the capturedcontextual environment, and may also determine placement informationfrom the placeholder information. The augmented reality application 124may retrieve an image from an item image source 122 of the server 120based on the placeholder information, placement information or anycombination thereof. In some embodiments, the augmented realityapplication 124 may retrieve the image from the item images 142 storedin a database 140 that is communicatively coupled to the server 120. Theaugmented reality application 124 may then modify the visual data toinclude image data associated with the retrieved item image. In someembodiments, the augmented reality application 124 may modify the visualdata with modified image data based on the determined placementinformation. In some embodiments, the augmented reality application 124then provides modified visual data to the processing system 110 forvisual display on a visual output 114.

In some embodiments, the processing system 110 may include an item imagesource, such as a library of item images stored on the processing system110. In an example embodiment, a user may create a library of images ofone or more items by way of the camera source 112 (e.g., digitalcamera). In one embodiment, an application hosted on the server 120 maybe used to identify an item depicted in the image (see U.S. patentapplication Ser. No. 12/371,882) and the augmented reality application124 may provide placeholder generation information from the server 120to the processing system. In some example embodiments, the processingsystem 110 may host the augmented reality application 124 and mayprovide modified visual data, such as to the processing system 110without use of the network 130 and the server 120.

FIG. 2 is a block diagram 200 of an example augmented reality module210, in accordance with one example embodiment. As shown in FIG. 2,augmented reality module 210 comprises in input receiver module 220, aninformation provider module 222, a visual data receiver module 224, adetection module 226, a placement determination module 228, a retrievalmodule 230, and a modification module 232. These modules of theaugmented reality system may be communicatively coupled to other modulesvia a bus 240.

An input receiver module 220 may be configured to receive a request tocommence an augmented reality application (e.g., augmented realityapplication 124 of FIG. 1). The request may include item informationassociated with a selected item. In one embodiment, the input receivermodule 220 may be configured to receive the request (e.g., an input)from a source, such as, for example, the processing system 110 ofFIG. 1. The input receiver module 220 may be configured to receiveadditional information from the processor system, such as, e.g., cameraspecifications, lighting and dimensional information of a capturedcontextual environment, data transmission rates, frame refresh rate,among other specifications of the processing system 110, and environmentparameters known to those of ordinary skill in the art.

An information provider module 222 may be configured to determineplaceholder generation information based on the received request anditem information from the input receiver module 220. Placeholdergeneration information may include instructions (e.g., machine readableinstructions) to generate a placeholder. In some embodiments, theplaceholder generation information may be machine-readable information,such as, for example, an electronic document. In an embodiment, theinformation provider module 222 may produce a file deliverable to therequesting source (e.g., processing system 110) or any other processingsystem. For example, the information provider module 222 may generate anelectronic document or an image file in any one or combination offormats (e.g., Portable Document Format (PDF), Microsoft® Word document,Joint Photographic Experts Group (JPEG) or the like). In operation, auser may interface with a processing system to select an item listing.The user may send a request to generate a placeholder for placementwithin an environment. The input receiver module 220 may receive therequest, which, in some embodiments, may include information identifyingthe item listing selected by the user, and the information providermodule 222 may generate an electronic document and deliver the documentto the requesting source (e.g., processing system 110). In someembodiments, the request to generate the placeholder may include theitem listing selected. In some embodiment, a user may receiveplaceholder generation instruction without selecting or sending arequest to generate a placeholder.

A visual data receiver module 224 receives visual data from a client,such as the processing system 110, communicatively coupled to a camera,such as a digital camera, video camera (e.g., web cam), or the like. Inone example embodiment, a user may position the camera to capture andview a contextual environment in which the user may want to view an itemas it would appear in that environment. In some embodiments, the visualdata receiver module 224 may process, some or all of the camera-capturedvisual data to determine parameters of the visual environment. Forexample, the visual data receiver module 224 may determine parameters ofthe captured environment including lighting, angles, dimensions,gravitational orientation, and the like, and may determine specifics ofthe camera, such as frame refresh rate, zoom capabilities, camera modelinformation, and the like.

A detection module 226 detects placeholder information that may beincluded with the received visual data. In an embodiment, theplaceholder information (e.g., machine-readable instructions to generatea document) is associated with a placeholder placed within a viewablelocation of a camera (e.g., within the camera-captured contextualenvironment), the camera communicatively coupled with a processingsystem (e.g., client device), the processing system providing visualdata received by the camera to the augmented reality module 210 at thevisual data receiver module 224. In one example, unique markings of theplaceholder may be captured by a camera coupled to the processingsystem, and these unique markings may be detected by the detectionmodule 226 as representing a placeholder in the captured environment.

In one embodiment, the detection module 226 may be configured to parsevisual data received from the visual data receiver module 224 todetermine whether the received visual data includes placeholderinformation within the visual data. In some embodiments, the detectionmodule 226 may verify the placeholder information, such as to determinewhether the placeholder information matches with the selected itemreceived at input receiver module 220. In some embodiments, thedetection module 226 may be configured to notify the client (e.g.processing system 110) when the placeholder information does not matchthe selected item. In an example, the detection module 226 may providesuggestions to the client source, such as to recreate the placeholder,reposition the placeholder, or the like.

A placement determination module 228 may be configured to determineplacement information of a placeholder using the placeholder informationreceived with the visual data. In one embodiment, the placementdetermination module 228 may determine various characteristicsassociated with the placeholder based on the placeholder information.For example, the placement determination module 228 may determinecharacteristics including coordinates and general location of theplaceholder in the contextual environment captured by the camera, andmay determine an orientation, scale factor, lighting effects, and anyobstructions obstructing the placeholder in the contextual environment,or the like. In one embodiment, the placement determination module 228may be configured to provide the placement information includingcharacteristics to a retrieval module 230. In one example embodiment,the placement determination module 228 may be configured to determinethe placement information of a placeholder based on unique markings ofthe placeholder. These unique markings may be detected by the detectionmodule 226 as described above.

A retrieval module 230 may be configured to retrieve an image of theitem upon detection of the image of the placeholder captured by thecamera. The retrieval module 230 may retrieve the image based on theunique markings of the placeholder. The retrieval module 230 mayincorporate information received from the detection module 226 and theplacement determination module 228 when retrieving an image of theselected item. In one example, the retrieval module 230 may select animage from a plurality of images of the item based on determinedcharacteristics of the placeholder determined from the placeholderinformation and received from the placement determination module 228, asdescribed above. In an example embodiment, the retrieval module 230 mayuse the determined characteristics to select a best fit image from aplurality of images. In some embodiments, the retrieval module 230 mayretrieve or determine physical dimensions of the item, such as theheight, the length, and the width, and may select a best fit image fromthe plurality of images according to the physical dimensions of the itemand/or the characteristics of the placeholder. In one example, theretrieval module 230 may be configured to modify image data associatedwith the retrieved image according to the physical dimensions of theitem and the characteristics of the placeholder, such as to represent ascaled and oriented image within the contextual environment.

A modification module 232 may be configured to modify the receivedvisual data to include image data associated with the retrieved imageinto the visual data according to the placement information. In someembodiments, the modification module 232 may be configured to modify thereceived visual data to include the modified visual data generated byretrieval module 230. In one example embodiment, the modification module232 may return the modified visual data to the client (e.g. processingsystem 110) providing the item selection and/or the selection to augmentreality.

It will be noted, that while FIG. 2 shows particular modules independentof other modules, other embodiments may include one or more modules in asingle component. Additionally, embodiments may be provided where acomponent that is shown in FIG. 2 as a single module maybe implementedas two or more components. Various operations performed by the augmentedreality module 210 may be discussed with reference to FIGS. 3-6.

FIG. 3 is a flow chart of a method 300 for providing modified visualdata, in accordance with an example embodiment. The method 300 may beperformed by processing logic that may comprise hardware (e.g.,dedicated logic, programmable logic, microcode, etc.) software (such asrun on a general purpose computer system or a dedicated machine), or acombination of both. In one example embodiment, the processing logic mayreside at a server 120 as shown in FIG. 1 in the form of an augmentedreality application 124 of FIG. 1.

As shown in FIG. 3, the method 300 commences at operation 310 when theserver 120 receives an input from a client (e.g., the processing system110 of FIG. 1), the input including an item selection. At operation 320,the method 300 may provide generation information for generating aplaceholder based on the received input. The generation information maybe provided to the processing system 110 to permit a user to generatethe placeholder using a printer, or view the placeholder using a visualdisplay device (e.g., a mobile phone screen, a camera screen or thelike). An example of a placeholder is depicted in FIG. 4 and describedbelow.

At operation 330, the method 300 receives visual data from the client(e.g., processing system 110). In one embodiment, the visual data may becaptured by a camera communicatively coupled with the client. Atoperation 340, the method 300 detects placeholder information includedwith the visual data; the placeholder information is associated with aplaceholder placed in a contextual environment captured by the camera.At operation 350, the method 300 determines placement information usingthe placeholder information. At operation 360, the method 300 retrievesan image of an item associated with the placeholder based on theplaceholder. At operation 370, the method 300 modifies the receivedvisual data to include the image data associated with the retrievedimage into the visual data with reference to the placement information.In an example, the retrieved image may be modified to reconcile therespective dimensions, orientation, etc. of the captured environmentwith the dimensions, orientation etc. of an item represented by theretrieve image. In some embodiments, the operation 370 of method 300 mayprovide the modified visual data to the client (e.g., processing system110) providing the input at operation 310. Now the user may view theitem as it would appear in the environment.

FIG. 4 is an example of a physical placeholder 400 for use with theaugmented reality system 100 (e.g. processing system 110, server 120,and network 130 all of FIG. 1), in accordance with an exampleembodiment. In one example, upon receiving an input including an itemselection, the augmented reality system 100 provides generationinformation to generate a placeholder 400 with unique markings 402-424to a client source from which the input originated or, in someembodiment, any other processing system (e.g., a mobile phone). Aspreviously described, the generation information may be a document,image, computer-readable instructions, or the like, for generating theplaceholder 400 by use of rendering instrument such as, for example, aprinter (e.g., in the case of a printable sheet) or a display screen(e.g., a display screen of a mobile phone, in the case of anon-printable digital file). In some embodiments, the placeholder 400may be a standard sheet of paper (e.g., dimension of about 8.5″×11″)with the unique markings 402-424. In other embodiments, the placeholder400 may be an electronically displayable set of instructions for displayon a mobile phone or an auxiliary display screen. In other embodiments,the placeholder 400 may be a three dimensional object.

The placeholder 400 may include any one or more of arrows 402, 404, 406,and 408, which provide the augmented reality system 100 an upside and adownside orientation for the placeholder 400, such as to orient theplaceholder 400 when captured by the camera. In one embodiment, theaugmented reality system 100 may calibrate a camera source 112 of FIG. 1of the processing system 110 with any combination of arrows 402, 404,406, and 408. In an example embodiment, the arrows 402, 404, 406, and408 may be formed in a clockwise direction such as to allow theaugmented reality system 100 to first determine a top side and bottomside orientation. When the page is flipped, the orientation determinedby the augmented reality system 100 may be flipped by 180°. In oneexample embodiment, any directionally distinguishable shape may besubstituted for one or more of the arrows 402, 404, 406, and 408. In oneembodiment, the augmented reality system 100 may differentiate betweenthe four corners of the placeholder 400 with use of the arrows 402, 404,406, and 408 when rotating the page across various axes from an originalorientation. In an example, when a user rotates the placeholder 400 inthe camera-captured environment, the augmented reality system 100detects the rotation and rotates the corresponding retrieved item imagewhen presenting the item in the contextual environment on a visualoutput 114 of FIG. 1.

In one example embodiment, the placeholder 400 includes a middlerectangle 410, which may provide for an initial detection of theplaceholder 400. In some embodiments, the augmented reality system 100may initially detect the middle rectangle 410 before other uniquemarkings when the placeholder 400 is placed a great distance away fromthe camera in an open environment, such as for example, an auditorium orin an outdoor environment. In other example embodiments, the middlerectangle 410 may be substituted with any design relatively large insize with relation to the placeholder 400 and generally located near themiddle of the placeholder 400.

As depicted in FIG. 4, the placeholder 400 includes shaded objects 412and 414 shaded with dissimilar colors to provide distinct regions of theplaceholder 400 for detecting an orientation, scale, and distance of theplaceholder 400. The augmented reality system may employ the shadedobjects 412 and 414 for determining a left-right and depth dimensionalorientation.

The placeholder 400 may further include line markings 416, 418, 420 and422 to assist with augmented reality system in refining a thirddimension for the placeholder 400. In one embodiment, the lines withineach of a first and second set of lines 416 and 418 are equidistant foreach set. The first and second sets of lines 416 and 418 may provide adepth of the object and a scale factor. By known methods, the augmentedreality system may orient and scale the image based on any or all of thearrows 402, 404, 406, and 408; middle rectangle 410; shaded objects 412and 414 and line markings 416, 418, 420, and 422.

In one embodiment, the unique markings of placeholder 400 may includeidentification markings 424, which may be item identificationinformation used when verifying whether the generated placeholder 400matches with a selected item. In other embodiments, these additionalidentification markings 424 may include or may reference additionalinformation related to the selected item, such as for example, price,inventory, and suggested items related to the selected item. In someexample embodiments, the additional information may be presentable alongwith the image of the item in the contextual environment. In someembodiments, the additional identification markings 424 may aid inuniquely identifying more than one placeholder 400 in the environment,and thereby, allow simultaneously presentation of a plurality of itemsin the contextual environment. In some embodiments, the identificationmarkings 424 may be a one-dimensional barcode, two dimensional barcode,or any other marking uniquely associable with a selected item. In someembodiments, a placeholder may be generated that may not be associablewith one particular item and may be interchangeably used to representmore than one item.

FIGS. 5A-B are screen shots 500A and 500B of example user interfaces502A and 502B illustrating a placeholder 550 in a camera-capturedenvironment and the resulting augmented reality image 560 displayed inthe contextual environment, in accordance with an example embodiment. Inan example, a user may direct a camera toward a location such as theuser's living room, which provides the contextual environment for theaugmented reality. The user interface 502A illustrates a visual frame504A of the camera-captured environment (e.g., a living room). In someembodiments, the user interface 502A and 502B may include selectionbuttons 506, 508, and 510. Selection button 506 may be selected toaugment reality, selection button 508 may be selected to purchase anitem, and selection button 510 may be selected to place a bid on anitem.

For example, a user may select an item, such as a lamp, and orient acamera to view an environment in which the user may want to place thelamp (e.g., a bedroom). The user may select the augment realityselection button 506 at a client (e.g., the processing system 110 ofFIG. 1) to provide an input along with the item selection information toa server (e.g., server 120 of FIG. 1). The server 120 may providegeneration information to the client (e.g., processing system 110) togenerate a placeholder, such as the placeholder 400 depicted in FIG. 4and the user may place the placeholder in the camera-capturedenvironment. Screen shot 500A illustrates a user interface 502A of FIG.5A with a placeholder 550 viewable within a camera-captured environmentin visual frame 504A.

The augmented reality application 124 of FIG. 1 at the server 120receives the visual data of the camera-captured environment (shown invisual frame 504A including the placeholder 550) viewable from theclient device (e.g., processing system 110). The augmented realityapplication 124 detects the placeholder information associated with theplaceholder 550 included with the visual data and determines placementinformation from the placeholder information. The augmented realityapplication 124 then retrieves an image of the item associated with theplaceholder 550. In some embodiments, the unique markings of theplaceholder 550 may be used to determine which item is associated withthe placeholder, or which image of the item may be selected from aplurality of images of an item. The augmented reality application 124provides the client device (e.g., processing system 110) with modifiedvisual data, which includes the image data associated with the retrievedimage of the selected item. In some embodiments, the augmented realityapplication 124 replaces the placeholder information with the imagedata, such as to modify the received visual data and to display areplacement image of the item within the camera-captured viewable invisual frame 504B. In an example embodiment, as shown in FIG. 5B, theaugmented reality application 124 may modify the received visual data byproviding overlay image data in place of the placeholder information,such as to display an overlay of the item image (e.g., augmented realityimage 560) within the camera-captured viewable in visual frame 504B.

FIG. 6 shows a diagrammatic representation of a machine in the exampleform of a computer system 600 within which a set of instructions, forcausing the machine to perform any one or more of the methodologiesdiscussed herein, may be executed. In alternative embodiments, themachine operates as a stand-alone device or may be connected (e.g.,networked) to other machines. In a networked deployment, the machine mayoperate in the capacity of a server or a client machine in aserver-client network, or as a peer machine in a peer-to-peer (ordistributed) network. The machine may be a personal computer (PC), atablet PC, a set-top box (STB), a Personal Digital Assistant (PDA), acellular telephone, a web appliance, a network router, switch or bridge,or any machine capable of executing a set of instructions (sequential orotherwise) that specify actions to be taken by that machine. Further,while only a single machine is illustrated, the term “machine” shallalso be taken to include any collection of machines that individually orjointly execute a set (or multiple sets) of instructions to perform anyone or more of the methodologies discussed herein.

The example computer system 600 includes a processor 602 (e.g., acentral processing unit (CPU), a graphics processing unit (GPU) orboth), a main memory 604 and a static memory 606, which communicate witheach other via a bus 608. The computer system 600 may further include avideo display unit 610 (e.g., a liquid crystal display (LCD) or acathode ray tube (CRT)). The computer system 600 also includes analpha-numeric input device 612 (e.g., a keyboard), a user interface (UI)navigation device 614 (e.g., a cursor control device), a disk drive unit616, a signal generation device 618 (e.g., a speaker) and a networkinterface device 620.

The disk drive unit 616 includes a machine-readable medium 622 on whichis stored one or more sets of instructions and data structures (e.g.,software 624) embodying or utilized by any one or more of themethodologies or functions described herein. The software 624 may alsoreside, completely or at least partially, within the main memory 604and/or within the processor 602 during execution thereof by the computersystem 600, with the main memory 604 and the processor 602 alsoconstituting machine-readable media.

The software 624 may further be transmitted or received over a network626 via the network interface device 620 utilizing any one of a numberof well-known transfer protocols (e.g., Hyper Text Transfer Protocol(HTTP)).

While the machine-readable medium 622 is shown in an example embodimentto be a single medium, the term “machine-readable medium” should betaken to include a single medium or multiple media (e.g., a centralizedor distributed database, and/or associated caches and servers) thatstore the one or more sets of instructions. The term “machine-readablemedium” shall also be taken to include any medium that is capable ofstoring, encoding or carrying a set of instructions for execution by themachine and that cause the machine to perform any one or more of themethodologies of embodiments of the present disclosure, or that iscapable of storing, encoding or carrying data structures utilized by orassociated with such a set of instructions. The term “machine-readablemedium” shall accordingly be taken to include, but not be limited to,solid-state memories, optical and magnetic media, and carrier wavesignals. Such media may also include, without limitation, hard disks,floppy disks, flash memory cards, digital video disks, random accessmemory (RAMs), read only memory (ROMs), and the like.

The embodiments described herein may be implemented in an operatingenvironment comprising software installed on a computer, in hardware, orin a combination of software and hardware.

Thus, an augmented reality system has been described. Althoughembodiments have been described with reference to specific exampleembodiments, it will be evident that various modifications and changesmay be made to these embodiments without departing from the broaderspirit and scope of the inventive subject matter. Accordingly, thespecification and drawings are to be regarded in an illustrative ratherthan a restrictive sense.

What is claimed is:
 1. A method comprising: visually capturing, by acomputing device, an environment including a placeholder, theplaceholder including a first portion and a second portion, the firstportion being visually distinct from the second portion; determining, bythe computing device, a placement of the placeholder in the environment,the placement describing an orientation of the placeholder and adistance of the placeholder relative to the computing device;identifying, by the computing device, an item corresponding to theplaceholder; retrieving, by the computing device, an image of the itemcorresponding to the placeholder; and generating, by the computingdevice, a visual display of the environment that includes the retrievedimage of the item, the retrieved image of the item being positioned andoriented in the visual display based on the determined placementinformation of the placeholder in the environment.
 2. The method ofclaim 1, further comprising receiving user input selecting the itemcorresponding to the placeholder and requesting the placeholder inresponse to receiving the user input.
 3. The method of claim 1, whereinthe visually capturing the environment including the placeholder isperformed using a camera of the computing device.
 4. The method of claim1, wherein the first portion of the placeholder includes at least twoshaded objects of dissimilar color.
 5. The method of claim 1, whereinidentifying the item corresponding to the placeholder is performed usingvisual information included in the second portion of the placeholder. 6.The method of claim 1, wherein the retrieved image of the item is scaledin the visual display of the environment based on the determinedplacement information of the placeholder in the environment.
 7. Themethod of claim 1, further comprising determining one or more physicaldimensions of the item, wherein retrieving the image of the itemcorresponding to the placeholder is performed in response to determiningthe one or more physical dimensions of the item.
 8. A system comprising:at least one processor; and a computer-readable storage media havinginstructions stored thereon that are executable by the at least oneprocessor to perform operations comprising: visually capturing anenvironment including a placeholder, the placeholder including a firstportion and a second portion, the first portion being visually distinctfrom the second portion; determining a placement of the placeholder inthe environment, the placement describing an orientation of theplaceholder and a distance of the placeholder relative to the computingdevice; identifying an item corresponding to the placeholder; retrievingan image of the item corresponding to the placeholder; and generating avisual display of the environment that includes the retrieved image ofthe item, the retrieved image of the item being positioned and orientedin the visual display based on the determined placement information ofthe placeholder in the environment.
 9. The system of claim 8, theoperations further comprising receiving user input selecting the itemcorresponding to the placeholder and requesting the placeholder inresponse to receiving the user input.
 10. The system of claim 8, whereinthe first portion of the placeholder includes at least two shadedobjects of dissimilar color.
 11. The system of claim 8, whereinidentifying the item corresponding to the placeholder is performed usingvisual information included in the second portion of the placeholder.12. The system of claim 8, wherein the retrieved image of the item isscaled in the visual display of the environment based on the determinedplacement information of the placeholder in the environment.
 13. Thesystem of claim 8, the operations further comprising determining one ormore physical dimensions of the item, wherein retrieving the image ofthe item corresponding to the placeholder is performed in response todetermining the one or more physical dimensions of the item.
 14. Acomputer-readable storage medium having instructions stored thereon thatare executable by a computing device to perform operations comprising:visually capturing an environment including a placeholder, theplaceholder including a first portion and a second portion, the firstportion being visually distinct from the second portion; determining aplacement of the placeholder in the environment, the placementdescribing an orientation of the placeholder and a distance of theplaceholder relative to the computing device; identifying an itemcorresponding to the placeholder; retrieving an image of the itemcorresponding to the placeholder; and generating a visual display of theenvironment that includes the retrieved image of the item, the retrievedimage of the item being positioned and oriented in the visual displaybased on the determined placement information of the placeholder in theenvironment.
 15. The computer-readable storage medium of claim 14, theoperations further comprising receiving user input selecting the itemcorresponding to the placeholder and requesting the placeholder inresponse to receiving the user input.
 16. The computer-readable storagemedium of claim 14, wherein the visually capturing the environmentincluding the placeholder is performed using a camera of the computingdevice.
 17. The computer-readable storage medium of claim 14, whereinthe first portion of the placeholder includes at least two shadedobjects of dissimilar color.
 18. The computer-readable storage medium ofclaim 14, wherein identifying the item corresponding to the placeholderis performed using visual information included in the second portion ofthe placeholder.
 19. The computer-readable storage medium of claim 14,wherein the retrieved image of the item is scaled in the visual displayof the environment based on the determined placement information of theplaceholder in the environment.
 20. The computer-readable storage mediumof claim 14, the operations further comprising determining one or morephysical dimensions of the item, wherein retrieving the image of theitem corresponding to the placeholder is performed in response todetermining the one or more physical dimensions of the item.